Graphically-based computer systems, for example mouse or pen-based computer systems, input data in a state that is visually distinguishable by the user of the input device. These same graphical input data, however, are often not distinguishable by the computer. Elaborate methods to recognize handwriting have been developed to attempt to overcome this problem.
The process of creating computationally distinguishable associations in material in graphically-based systems without computationally recognizing the material has been very difficult. The material may be of any sort: images, text, graphics, or recorded material. It is desirable to apply category indices to portions of material to label them for later retrieval of relevant portions. Prior methods of annotating or categorizing material in real time using computationally recognizable indices may require the user to either 1) click on and off a preset list of categorizers, 2) attempt to type real-time notes to take advantage of the system's ability to computationally recognize typed text, or 3) manually correlate notes, adding computationally significant markers, at a later time. In all of these cases, users are forced to comply with the computational limitations of the system rather than have a technique which adapts to their natural method of notetaking. In order to make material more accessible, users require a facile method for creating a set of personally structured annotations. and retrieving annotations without undue computational effort.
Other proposed methods for categorizing material for later retrieval provide pre-defined index categories, with some kind of menu or palette of these pre-defined indices provided to the user, along with some kind of technique for attaching the indices to the portions. For example, the indices can be arrayed in a button panel alongside a window displaying the "current" portion of material; and indices could be attached to the current portion by "touching" relevant buttons.
A problem with this method is that pre-defined categories are not usually sufficient. The user will want to create categories to better characterize the material at hand, and "ad hoc" categories will emerge during the process of examining the materials. It is possible to have the user create ad hoc categories by typing in names for them.
Giving users sets of pre-made categorizers to apply to the material may force the users to operate sets of buttons or invoke commands while at the same time trying to digest the content information, figure out which category it belongs to, as well as take notes. The button method also requires that the user remember to deactivate a button, as failure to do so provides a false structure to the categorization.
Computer systems may be used for notetaking during events, which may also be recorded simultaneously, or pre-recorded. Text-based systems using keyboards may be used in these instances, but since most people talk much faster than they type, creating computer generated textual labels to describe the content in real time requires enormous effort. Additionally, the click noises of the key strokes may be distracting to the user or others present, and may contaminate audio tracks of recordings made of real time events. To aid in solving some of these problems, pen-based systems have been applied.
In pen-based notetaking and recording systems, locating interesting parts of a recorded media may be assisted by a correlation between the notes and the time base of the recording. For example, Lamming, EP-0 495 612 describes a notetaking system based on a notepad-size computer with a graphical input device. Each stroke and character input by the user is given a time stamp. When a stroke or character is later selected, the recorded media correlated to the time stamp of that particular object is retrieved. By entering new, separate marks near a previously entered indicium whenever an idea or speaker or topic applies to the previously entered indicia, later selection of all of the marks spatially associated with that indicia will result in all sections of the recording indexed by the time stamps of the respective marks to be replayed.
Rindfuss, U.S. Pat. No. 4,841,387 presents a pen-based system for recording and indexing information, in which the positions of handwritten notes are stored in a correlation table correlated with the position of the recording medium. When a portion of the handwritten notes is later identified, the system uses the correlation table to locate the corresponding portion of the recorded information.
In the pen-based computer and recording systems above, the computer must correlate the time a particular pen mark is made with the portion of the recording made at that time. In graphically-based systems, where there is no recording made or available, these time-based marks may have little information that is inherently useful for categorizing relationships in the graphical information without a pre-thought-out plan provided by the user. The user's ability to index notes is entirely controlled through the time stamps of the indicia the user has entered in a document. In the case of Lamming, this time-stamping does not allow for later, real-time augmentation of previously entered notes while the event continues, with additional notes or categorizations related to previously entered notes added in real time as the event is taking place, since later notes will be time-stamped with the time they were entered rather than with the time of the material to which they relate.
With respect to a document not related to a recorded event, and that contains text or graphics, a user may wish to indicate or retrieve relationships in different parts of a document. However, without sophisticated handwriting recognition programs, or a related keyboard system, most forms of annotation in pen-based systems are not computationally recognizable to the computer.
All that is actually needed in a graphically-based system, however, are visually distinguishable icons which allow the user to tell one from another and to remind the user of what the icon means. These icons may be applied to different portions of data structures to establish a relationship between the portions, and may be used to retrieve associated data objects based on the relationships. The computer system need only be able to computationally distinguish between one icon and another, without necessarily distinguishing the meaning of the icon. The computer may then distinguish and present a computationally significant relationship between objects tagged with the same icon.
It is an object of the present invention to provide a rapid and facile method for defining categorical objects and using them to denote categorization of other objects in a graphically-based computer system. This object is obtained by allowing the user to create visible icons which are assigned unique identifiers when they are created. The visual icons are used as a reference to attach the distinguishable identifiers to other data objects, establishing computationally significant relationships between the visible icons and the data objects, and between data objects with similar identifier attachments.
It is a further object of the invention to provide a rapid and facile method for manipulating categorical information about uninterpreted data objects in a graphically-based computer system. This object is obtained by providing visual representations of the associations between selected data objects and categorizational objects, and providing an ability to change the associations based on graphical representation data.